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PREAMBLE 


The Provincial Auditor (the Auditor) reported on Science and Technology in 
Section 3.07 of his 2003 Annual Report. The Standing Committee on Public 
Accounts held hearings on this audit report on February 17, 2004, with 
representation from the Ministry of Economic Development and Trade. (At the 
time of the audit, these services fell under the Ministry of Enterprise, Opportunity 
and Innovation.) 


The Committee endorses the Auditor’s 2003 report on Science and Technology 
and recommends the implementation of his recommendations by the Ministry of 
Economic Development and Trade. The Committee has prepared supplementary 
recommendations based on its findings during the hearings. This report is a record 
of those findings and the Committee’s recommendations. 


The Committee extends its appreciation to the officials from the Ministry for their 
attendance at the hearings. The Committee also acknowledges the assistance 
provided during the hearings by the Office of the Provincial Auditor, the Clerk of 
the Committee, and staff of the Legislative Library’s Research and Information 
Services. 


Ministry Response to Committee Report 


The Standing Committee on Public Accounts requests that the Ministry of 
Economic Development and Trade provide the Committee Clerk with a written 
response within 120 calendar days of the tabling of this report with the Speaker of 
the Legislative Assembly, unless otherwise specified in a recommendation. 


Committee Comment 


The Committee wishes to commend the Ministry of Economic Development and 
Trade on their commitment to responding to the recommendations found in the 
Auditor’s 2003 Annual Report respecting its science and technology programs. 


1. OVERVIEW 


The former ministries of Energy, Science and Technology, and Enterprise, 
Opportunity and Innovation funded several major science and technology 
programs and spent $1.3 billion between April 1, 1998 and March 31, 2003. The 
government announced total program commitments of $4.3 billion over the same 
time period. ! 


2. AUDIT OBJECTIVES AND SCOPE 


The audit’s objectives were to assess whether the Ministry had adequate 
procedures in place to: 





ensure compliance with Management Board of Cabinet directives, contractual 
agreements, and Ministry policies; 

ensure that resources were managed with due regard for economy and 
efficiency; and 


measure and report on the Ministry’s effectiveness in fulfilling its mandate 
and identifying areas where corrective actions are required. 


The audit was substantially completed by March 2003 and included an 
examination of six science and technology programs that accounted for 90% of 
the Ministry’s science and technology expenditures. 


2.1 Access to Information 


The Auditor’s staff did not receive adequate access to all the information they 
requested from the Ministry, information that must be provided to them under s. 
10 of the Audit Act. These limitations meant that they were prevented from 
concluding on their audit objectives and completing the audit in a timely manner. 
The Ministry was of the opinion that it provided the information in accordance 
with government protocols.’ 


Committee Hearings 


Standard protocols were followed, according to Ministry staff. There was no 
intent to delay or withhold the information requested. Unintended delays were due 
to internal processes and the breadth of the requests. The Auditor was given 
assurances by the Ministry that it will do better in the future. Processes have been 
modified and streamlined to ensure future requests will be met more quickly and 
fully.* This was also the first audit of science and technology programs. The 
situation was further complicated by the fact that programs had just been moved 
from one ministry to another.° 


3. OVERALL AUDIT CONCLUSIONS 


Because of the previously mentioned limitations on access to information, the 
Auditor cannot provide reasonable assurance that the audit identified all matters 
that should be brought to the attention of the Legislative Assembly. The review of 
the Ministry’s science and technology spending of $1.3 billion between April 1, 
1998 and March 31, 2003 did, however, reveal a number of significant concerns. 
A major concern was the commitment to spending $4.3 billion without an overall 
strategic plan to set parameters and consistent policies for existing programs or to 
guide the development of new programs to meet the objectives of promoting 
Innovation, economic growth and job creation.° 


4. COMPLIANCE WITH GOVERNMENT AND MINISTRY POLICIES 


4.1 Governance and Accountability 


Ontario Innovation Trust 

The Ontario Innovation Trust (the Trust) was created in March 1999 to support 
the capital costs of research conducted in universities, colleges, hospitals, and 
research institutions. It was to match grants from the Canada Foundation for 
Innovation (CFI) and complement the Ontario Research and Development 
Challenge Fund, which funded research operating costs. A board was appointed to 
oversee the Trust. A subsidiary, the Innovation Institute of Ontario (IIO), was 
created in 2000 to administer grants to research institutions. The Ministry has 
committed more than $1 billion to the Trust. As of March 31, 2003, $750 million 
had been paid. The Trust had more than $500 million on hand as of March 31, 
2002." 


The Trust was created by an agreement between the former Minister of Energy, 
Science and Technology, and a private-sector corporation designated as the 
Trustee. Contrary to government directives on alternative service delivery, there 
was no submission to Cabinet and no business case to justify the Trust’s creation. 
Contrary to the directive on transfer payment accountability, there was no 
agreement between the Ministry and the Trust defining provincial expectations or 
outlining how the Trust would be held accountable for its spending of public 
funds. 


Although required, the Ministry receives virtually no information from the Trust. 
The required monitoring process is not in place to ensure compliance with the 
agreement. The Trust is not required to report to the Minister or through him/her 
to the Legislature. Its first annual report was not released until three years after its 
creation. 


The Lieutenant Governor in Council appoints three of the Trust’s seven board 
members, none of whom are Ministry staff. At the time of the audit, no one had 
been named to replace one government appointment that had expired in May 2001 
and another that had expired in May 2002. (In addition to the members appointed 
by the Lieutenant Governor-in-Council, two are appointed by the Council of 
Ontario Universities and one each by the Ontario Hospital Association and the 
Association of Colleges of Applied Arts and Technology.’) 


The Auditor concluded that neither the Legislature nor the Ministry has any 
control over the spending of taxpayers’ funds by the Trust. He recommended that 
the Ministry negotiate an agreement with the Trust to establish proper governance 
and accountability arrangements, implement procedures for routinely obtaining 
and reviewing information on the status of recipient eligibility and ongoing 
results, implement procedures for identifying areas of non-compliance and 


* As of 17 February 2004, the Trust had committed $708 million to over 1,000 projects at 45 
institutions across the province. See Ontario, Legislative Assembly, Standing Committee on 
Public Accounts, Official Report of Debates (Hansard), 38th Parliament, 1st Session (17 February 
2004): P-142. 
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initiating corrective action where required, and ensure that all government board 
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appointments are up to date. 


Committee Hearings 


The Auditor's report was credited with helping to address concerns arising from 
the arm's-length structure of the Trust, as well as the Trust agreement’s limitations 
on the Ministry’s capacity to work directly with it. Committee members were told 
that the Trust board had agreed to adopt the same processes as transfer payment 
agencies with respect to accountability issues. It has committed to tabling annual 
financial statements’ and annual reports in the Legislature, appearing before 
legislative committees, and making records available to the Auditor. At the time 
of the hearings, the Ministry was working on an accountability framework to 
formalize these arrangements. The target date for completion was no later than 
July 2004. The framework will be set out in a memorandum of understanding 
(MOU) between the Trust board and the Ministry. (A month after the hearings, 
the Ministry wrote the Committee to say that an MOU with the OIT was expected 
to be “executed” by the end of May 2004 instead of July.’) 


The Ministry was in the process of preparing the documentation for confirming 
three new government Order-in-Council board appointments. ! 


Trustee, Sponsor and Board 


The Trust was established by an agreement with a financial institution, the 
Trustee. The original Trustee was the Royal Trust Corporation of Canada. There 
is provision in the agreement for that to change, but Ministry staff were unable to 
say which financial institution is now Trustee. The Trustee is the caretaker of the 
Trust’s monies and distributes them at the direction of the board. 


The government is the Trust’s sponsor but is excluded from any involvement with 
it other than having the capacity to appoint three board members. The sponsor has 
no relationship with the Trustee. 


At the time of the audit, there were two vacancies among the three government- 
appointed board members. During their presentation to the Committee, Ministry 
staff said that documents were being prepared for confirming three new 
government appointments. When asked how long the positions had been vacant, 
they replied that any one could have been vacant for as long as two or three years. 
The fact that terms were not extended or that people were not replaced did not 
result in any failure in the operation of the board. The Trust agreement states that 


individuals continue to have the duties and obli gations of a member until a 
successor is appointed.'! 


The government appointees to the board were thought to have deputy minister 
status. While there is no formal documentation of a reporting obligation, Ministry 
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Sue staff thought these statements would be for the Trust as opposed to each project. See 
tanding Committee on Public Accounts, Official Report of Debates (Hansard), p. P-162. 





staff assumed that these appointees would have a corporate responsibility to 
government. if 


Choice of Structure and Accountability 


Ministry staff declined to speculate on how the decision was made to structure the 
Trust in its existing form. They did allow that the choice was influenced by the 
CFI’s arm’s-length framework. (Unlike the Trust, the CFI was established under 
legislation.) Because one of the primary intents of the Trust was to match CFI 
funding for Ontario institutions, a similar operating structure would facilitate that 
process. The choice of structure was referred to as an accepted and appropriate 
way to fund R&D infrastructure. It also allowed for flexibility which might 
improve the availability of longer-term partnerships. 


There was no submission to Cabinet or business case presented because the 
government wished to move quickly to take advantage of CFI funding. That 
money is distributed across the country and involves periodic calls for proposals. 
If an institution is unable to attract funding from its provincial government, there 
is a risk of not receiving CFI funding. 


Ministry staff were unaware of a project involving a similar funding commitment 
being introduced without a submission to Cabinet. They did, however, qualify 
their response by stating it could have happened. '° | 


Within the chosen structure, Ministry staff were able to confirm that the Trust was 
adhering to rigorous peer review processes when evaluating applications. (Many 
projects would have already been through the CFI peer review process.) Regular 
reports were published on the Trust’s web site. An informal working relationship 
between the Trust and the Ministry had developed over time and kept the latter 
aware of Trust activities, particularly as they related to other Ministry-managed 
science and technology programs. 


With respect to the timing of the accountability mechanisms they had said would 
be put in place, Ministry staff reported that work had begun on a process two or 
three months before. It was their hope not to appear before the Committee again 
about the Trust after reference was made to the Committee’s 2002 report, Ontario 
Innovation Trust.'® When asked if they would follow a different course of action 
given a similar situation in the future, Ministry staff replied that an array of 
options would be presented, one of which would be a trust." 


Flow of Funding 


The Trust has a 10-year lifespan. The government’s initial financial commitment 
was augmented by an announcement in a subsequent budget. In addition, the 2002 
budget included a $300 million enhancement to the Trust. (At the time of the 
hearings, the $300 million had yet to be confirmed.) Interest earned by the Trust is 
also available for project funding. 


The Trust manages approved projects through to completion; most are capital so 
expenditures can occur over several years. Ministry staff expected that all 





commitments would be made well within the 10 years of the Trust. When asked if 
the initiative could become oversubscribed, they felt there was a possibility of that 
occurring by April 2005 in terms of applications relative to cash available. 


In the case of an approved project that ‘derails’ during the life of the Trust, 
funding remains with or is returned to the Trust. The Trustee will distribute any 
money left in the Trust at the end of the 10 years among the Trust’s 
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beneficiaries. 


In response to a request from the Committee, the Ministry later reported on the 
schedule of payments made to the OIT.*' 


Date of Payment Amount ($) 
March 31, 1999 10,000 
April 30, 1999 249,990,000 
May 26, 2000 500,000,000 
Total 750,000,000 


At the same time, the Committee learned that the Trust had earned $90 million in 
interest (less expenses and administrative costs) as of December 31, 2003.7” 


Committee Recommendations 


The Ontario Innovation Trust’s (the Trust’s) purpose and intent are laudable and 
have the potential to be of great benefit to the province as a whole. While the 
Trust’s board of directors has responded positively to the accountability issues 
raised by the Auditor, the Committee has concerns with respect to the precedents 
its creation may have set. 


The Committee therefore recommends that: 


1. The Ministry of Economic Development and Trade must give due 
consideration to oversight, compliance and governance implications 
before creating other bodies like the Ontario Innovation Trust. 


2. The Ministry of Economic Development and Trade report on the 
accountability measures contained within its memorandum of 
understanding with the Ontario Innovation Trust. 


The Committee requests that the Ministry provide the Committee 
Clerk with a written response within 30 days of the tabling of this 
report in the Legislature. 


Ontario Research and Development Challenge Fund 

The Ontario Research and Development Challenge Fund (Challenge Fund) was 
established in 1997 as a 10-year Program to promote research of interest to the 

private sector, and to encourage collaboration between that sector and research 
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institutions. It was also designed to improve Ontario’s ability to attract and retain 
researchers and help universities compete for federal funding. More than 100 
projects had been approved as of March 2003, for a total Ministry commitment of 
$435 million. Sponsoring research institutions and private-sector organizations 
had made additional commitments of $437 million and $510 million, respectively. 
A Challenge Fund board was created to provide independent advice and 
recommendations to the government on research and development proposals. The 
government was represented on the board by five ministries. A 1997 Cabinet 
submission established the former Ministry of Economic Development, Trade and 
Tourism as the ministry responsible for administering the Challenge Fund and 
being accountable to the Legislature. The ministries and the board signed an 
MOU in August 1999. It was to be formally reviewed every two years. The first 
MOU review had not been completed as of March 2003, even though significant 
changes had occurred in the Challenge Fund (e.g., delegation of some 
responsibilities to the Innovation Institute of Ontario - IIO). 


The MOU permits the board’s chair to appoint special advisory committees to 
assist in evaluating proposals. Non-medical and medical panels had been 
established by the time of the audit, but neither had terms of reference outlining 
their roles and responsibilities. In addition, the MOU does not establish 
performance targets or designate a lead ministry. 


The Auditor recommended that the Ministry update the MOU between the 
Challenge Fund and the ministries to outline the responsibilities of the Challenge 
Fund board and special advisory committees, and to reflect current program 
objectives and suitable performance measures, and ensure that primary and direct 
oversight responsibility for the Challenge Fund rests with a lead ministry.” 


Committee Hearings 


The Auditor's concerns had provided strong direction on strengthening 
accountability processes for the Challenge Fund program. The government's 
almost $480-million commitment had leveraged $1.2 billion from research 
institutions and their business partners. 


The IIO is under contract by the Ministry to support the administrative function of 
the Challenge Fund. It is working with the Challenge Fund board to update the 
MOU originally signed on March 2, 1998 by the five ministries involved in the 
establishment of the Challenge Fund - currently four ministries, due to the 
realignment of science and technology. 


The framework for the new MOU was developed in December 2003 and was 
being refined in draft form. It will include designation of the MEDT as the lead in 
managing the program. It will clarify program objectives and define the respective 
roles and responsibilities of the parties to the MOU, the MEDT and the board. 
Work on the new MOU was expected to be complete by the end of 2003/04. 
Some of its provisions were already being implemented.” 
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4.2 Project Selection 

Ontario Research and Development Challenge Fund 

The IIO has administered the Challenge Fund since November 2000. The IIO 
reviews and analyzes grant proposals to ensure complete documentation ISS, gs 
submitted and that eligibility criteria are met. It also prepares project summaries 
for use by the board. With the board, the IIO determines which proposals require 
expert assessments. After a panel reviews proposals, the IIO summarizes its . 
recommendations and forwards them to the board. The board’s recommendations 
are forwarded to the five ministers for final approval. 


The Auditor’s staff reviewed the selection process and found that procedures were 
not in place to ensure that only eligible projects were selected and that the process 


was fair and transparent. 


The Auditor recommended that the Ministry require the IIO to retain all relevant 
documentation; implement procedures for periodically verifying eligibility and 
ensuring that any exceptions to program eligibility criteria are well supported; 
ensure that all applicable ministers are apprised of the Board’s recommendations 
or, if appropriate, obtain a delegation of authority for the Minister of Enterprise, 
Opportunity and Innovation to approve projects for funding on behalf of all 
ministers who are party to the agreement; and ensure that applications for research 
funding are reviewed within the specified time frame and that recommendations 
are made to the required ministers on a timely basis.”° 


Committee Hearings 


The Ministry was working with the IIO to address the issues raised by the Auditor 
and expected to have this work implemented by the end of March 2004.”° 


Ontario Research Performance Fund 

The Ontario Research Performance Fund (Performance Fund) was introduced in 
2000 to provide funding for institutions conducting Ontario government- 
sponsored research. It contributes to the overhead costs of eligible projects at 40% 
of direct costs. The rate is reduced when overhead support is received from other 
ministries. In 2001/02, 218 projects received funding. In its first two years, the 
Performance Fund was to spend $30 million annually. If total overhead funding 
exceeded this amount, each recipient’s funding was to be reduced to ensure total 
annual payments did not exceed $30 million. 


Institutions designated as eligible research institutions in a provincial tax bulletin 
are deemed eligible for funding. Those not listed can apply for an assessment of 
their status. The audit found that the Ministry did not have documentation to 
demonstrate the process used to approve the eligibility of 41 new recipients not 
listed in the bulletin. These recipients received $8.7 million in 2001/02. 


Eligibility is also contingent on a signed confirmation letter from the lead ministry 
that provided the initial research funding. The amount of the initial grant for direct 
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costs is used to calculate the overhead grant. Confirmation letters were not on file 
for a number of projects, meaning the Ministry did not always have sufficient 
documented evidence regarding the amounts used to calculate grants. 


Grants reimburse overhead costs for the preceding year. Institutions must submit a 
performance report outlining the use of a grant in the year in which it is received. 
The Auditor’s report noted that this reporting requirement did not meet its 
intended purpose. 


There is no deadline for eligible recipients to submit funding requests. This means 
submissions can be received after all of the annual funding has been paid out and 
late submissions are paid in subsequent fiscal years. In 2000/01 and 2001/02, no 
adjustments were made to amounts paid to other recipients, resulting in 
overpayments or payments in excess of $30 million in each year. 


The Auditor recommended that the Ministry ensure that all new recipients meet 
program eligibility requirements; ensure that signed confirmation letters are on 
file verifying the amount of grants approved by other ministries to eligible 
recipients; implement procedures for verifying that grant amounts are calculated 
accurately; and establish a deadline for submissions and for finalizing annual 
payments under the pro gram.’ 


Committee Hearings 


The now $32-million annual fund was set up following a 1999 report, Growing 
Ontario's Innovation System: The Strategic Role of University Research. A key 
recommendation was that the government establish a research performance fund 
to resource universities for the indirect costs associated with increasing their 
research capacity. 


According to Ministry staff, implementation of the recommendations regarding 
the administration of this program is complete, except for one activity which is 
still in the final stages of being amended. Recommendations already implemented 
include the retention of confirmation letters, clearer verification of grant amounts 
and deadlines for confirmation of information by participating ministries and 
institutions, with payments. 


The Ministry was finalizing an administrative procedures manual so that it can 
document internal procedures and payment processing for the Performance Fund 
grants. This was expected to be complete by March 2004 together with a plain 
language text for general use.”® 


Premier’s Research Excellence Awards 

The Premier’s Research Excellence Awards (PREA) program was established in 
1998 to attract and retain graduate students and researchers. Awards are made on 
a competitive basis and paid to principal researchers’ institutions. A volunteer 
advisory board reviews all applications and makes recommendations to the 
Minister. Each recipient is eligible for up to $100,000. An additional 50% from 
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either a research institution or a private-sector partner is required. Approximately 
$45 million was awarded to 438 recipients, as of March 31, 2003. 


Potential recipients must submit a standard application, letters of reference and 
letters guaranteeing the 50% matching private-sector contribution. Applications 
are distributed to pairs of board members for review. The entire board considers 
10 applications recommended by each pair of members. (The Chair reviews them 
all.) Complex applications may also undergo an external peer review. Score 
sheets and ratings are forwarded to the Ministry which summarizes the score 
sheets and prepares a short list. This list is considered by the board which 
forwards its recommendations to the Minister for approval. 


The audit found that the Ministry had not maintained sufficient documentation to 
support selection decisions. This meant it was not possible to verify whether the 
most deserving applicants were those recommended for awards. 


The Ministry’s Internal Audit Services Branch surveyed recipients following the 
first round of competition and found the selection process and program criteria 
were unclear. The Auditor’s staff reviewed more recent competitions and found 
the evaluation criteria, scoring process and weighting for each criteria continued 
to be unclear. This means there is no assurance that the scoring criteria for all 
proposals are comparable. Other inconsistencies in the evaluation process 
included a different rating system for new researchers. 


The Auditor recommended that the Ministry ensure that all selection documents — 
including proposals, individual score sheets, summary score sheets, and written 
recommendations ~ are kept on file for a specified retention period; all individual 
and summary score sheets are reviewed for accuracy; and the selection process 
and evaluation criteria are explicitly stated to potential applicants and applied 
consistently.” 


Committee Hearings 


The Auditor’s recommendations on administrative procedures regarding record- 
keeping and document retention during the award selection process were being 
acted upon. The Ministry was also revising program information to ensure that all 
potential applicants are fully informed about evaluation and selection criteria and 
procedures, and their transparency. Ministry staff had met with the PREA 
advisory board in January 2004 and would work closely with it to implement the 
program in accordance with the revised procedures.°? 


Contained within the Auditor’s report was the finding that the marks on the 
reviewers’ individual score sheets for one round of competition did not match the 
Ministry’s summary score sheet, The marks on the summary score sheet did not 
match the summary of scores on the short list of recipients recommended for 
funding. The Committee heard that PREA applicants come from a broad range 
of subject backgrounds. As applications made their way through early review 
processes, the sense of weighting may have changed as the merits of each 
Initiative were discussed. The program officer for PREA had assured Ministry 
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staff Rigen at the hearings that administration processes were much tighter 
now. 


Committee Recommendation 


The Committee recommends that: 


3. The Ministry of Economic Development and Trade report to the 
Committee on its progress in improving the Premier’s Research 
Excellence Awards’ selection process procedures. 


The Committee requests that the Ministry provide the Committee 
Clerk with a written response within 120 days of the tabling of this 
report in the Legislature. 


Premier’s Platinum Medal for Research Excellence 

The Premier’s Platinum Medal for Research Excellence (Premier’s Platinum 
Medal) was created in 2001 to keep outstanding senior researchers in Ontario. It is 
intended for researchers at the mid-career level who have an international 
reputation and a significant body of research. The program is to cost $10 million 
over six years. The award includes a medal, a citation and $1 million to support 
the winner’s research at an eligible research institution. 


Responsibility for overseeing the program was assigned to the PREA advisory 
board. A 2001 Cabinet submission indicated the Ministry would amend its MOU 
with the board to outline terms of reference, the board’s mandate and performance 
measures. No changes had been made to the MOU as of March 31, 2003. 


Peers nominate potential recipients. A selection subcommittee is drawn from the 
PREA board, and the Ontario Science and Innovation Council. It conducts an 
initial assessment to create a short list of seven nominees which is forwarded to 
six international reviewers. The PREA board reviews the reviewers’ assessments 
and then recommends two nominees to the Minister. The first two awards had 
been announced but no payments had been made at the time of the audit. 

A fair and transparent selection process is essential to this type of program. 
Among the problems with the process identified by the audit were: incomplete 
and missing score sheets, undated nomination packages, and inadequate 
documentation to support the selection of seven shortlisted candidates. 


The Auditor recommended that the Ministry update the MOU with the advisory 
board to reflect the board’s responsibility for the program, and retain all 
documentation necessary to adequately support the eligibility and selection of 
each recipient of the Premier’s Platinum Medal for Research Excellence.” 


Committee Hearings 


The Ministry was updating its MOU with the PREA advisory board to reflect the 
board's responsibilities with respect to the Premier's Platinum Medal. It noted its 
commitment to having new administrative policies and procedures, including 
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rigorous observance of deadlines, transparency of nomination review and award 
selection, and an enhanced document retention schedule in place before the next 


round of the Premier's Platinum Medal.** 


4.3 Program Monitoring 


Ontario Research and Development Challenge Fund Grants 

The transfer payment accountability directive requires that program managers 
obtain and review information on the status of recipient eligibility and 
performance. They must also ensure that all required reports are received when 
due, and reviewed and analyzed on a timely basis. According to the accountability 
directive, the obligation to answer for results and for how responsibilities are 
discharged cannot be delegated to other parties. 


The Challenge Fund requires research institutions to submit quarterly 
disbursement requests, annual progress reports, audited reports at various stages, 
and a final report when the project is complete. The required reports are submitted 
to the IIO, the Challenge Fund’s administrator. 


The audit reviewed this monitoring process and noted a number of 
inconsistencies. For example, the IIO was administering approximately 60 
projects listed as active, but almost half had not received funding for more than 
six months. A sample of active projects found that only about 10% had submitted 
the required annual report on time. There was no overall policy regarding when 
audited reports are required. Only three of 15 completed projects had submitted 
their final reports on a timely basis. The other 12 were submitted late or remained 
outstanding. 


The Auditor recommended that the Ministry review for continued eligibility all 
projects that have not received payments from the Challenge Fund in the previous 
six months, and implement an ongoing process for identifying and following up 
on such projects; establish an overall policy regarding when audited reports are 
required, implement clear guidelines on the form and content of these reports, and 
ensure that quarterly, annual, audited, and final project reports are received when 
due; and on a timely basis review and analyse all reports received to ensure that 
projects remain eligible, to determine whether milestones have been met, and to 
assess whether performance has been satisfactory.*> 


Committee Hearings 


The Ministry was working with the IIO to update policies and procedures for 
financial and program reporting, and for follow-up to take corrective action when 
necessary. For example, it has received and is following up on a quarterly report 
on those Challenge Fund projects that have not received funds in six months 
because of delays in project implementation. Mitigation measures were being put 
in place to address undue delays in project implementation.*° 


Other initiatives being developed with the IIO include project detail reports which 
will capture titles, descriptions, reporting procedures, and the timeliness of the 
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receipt of reports. Administrative processes and procedures will be more tightly 
defined at the beginning of a project cycle. 


The government’s conversion from modified cash to accrual accounting will place 
different obligations on the Ministry as program manager with respect to financial 
flows to projects.” The new contractual arrangements that will result from the 
conversion will require more attention be paid to timing and deliverables so that 
project budgets can be managed within a more contained time frame.’ 


Committee Recommendation 


The Committee recommends that: 


4. The Ministry of Economic Development and Trade report on its 
efforts and those of the Innovation Institute of Ontario to update 
policies and procedures respecting financial and program reporting 
for Ontario Research Development Challenge Fund grants. 


The Committee requests that the Ministry provide the Committee 
Clerk with a written response within 120 days of the tabling of this 
report in the Legislature. 


Ontario Centres of Excellence 

The Ontario Centres of Excellence (OCE) program was established in 1987. It 
provides industry with access to leading-edge ideas, and helps transform those 
ideas into products and services. There are four Centres of Excellence: Materials 
and Manufacturing Ontario; Communications and Information Technology 
Ontario; Photonics Research Ontario; and the Centre for Research in Earth and 
Space Technology. Annual funding to the four centres is approximately $33 
million. 


More than $500 million has been provided to the OCE program since it was 
established. Management Board directives require the Ministry to implement an 
effective monitoring process. In the case of the OCE program, this means the 
Ministry must receive performance reports from each centre. It requires the 
submission of annual operating plans, audited financial statements, and reports 
from the president. 


The audit’s review of the monitoring process noted that while three of the four 
operating plans for 2002/03 had been submitted late, all were eventually 
submitted and included all of the major requirements. Audited financial 


* “Under accrual accounting, the accounts record a transaction in the year in which the Province 
takes on a legal obligation to pay an outside party, or the outside party takes on an obligation to 
make a payment to the Province. This applies whether the transactions are actually finished by the 
end of that year. Instead of waiting until all the cash flows finish, the Province’s accountants will 
now estimate what the flows will be after the end of the year. This is a normal part of accrual 
accounting.” See Ontario, Ministry of Finance, Financial Report 1994 (Toronto: The Ministry, 
October 1994), p. 4. 
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statements had been received from all four centres for 2001/02. While none of the 
centres had ever submitted a president’s report, all had submitted annual reports 
that contained some elements of a president’s report. At the time of the audit, 
March 2003, none of the centres had submitted an annual report for 2000/01. 


The Auditor recommended that the Ministry implement a process for tracking the 
receipt of all required monitoring reports, and follow-up on any outstanding 
reports in a timely manner; and adequately review all reports received and 
reconcile the annual reports’ information with that contained in the audited 
financial statements to ensure that reported information is accurate and 


complete.*® 


Committee Hearings 


In order to enhance the strategic alignment of programs to government economic 
and innovation policies, the Ministry has worked with the existing centres to 
restructure the OCE program’s governance with the creation of a single 
organization - Ontario Centres of Excellence Inc. - through which the four current 


centres will be operated. 


As the Ministry finalizes its new contract for the operations of the OCE program, 
it will incorporate the Auditor's recommendations to ensure that appropriate 
reporting, program monitoring and accountability measures are put in place and 
observed. The new contract was to be ready for execution by the end of March 
2004. Once it is finalized, a report tack system will be established, with staff 
reviewing and analyzing received reports.” 


Premier’s Research Excellence Awards 

Institutions whose researchers receive grants from the PREA program are 
required to submit annual financial and performance reports to the Ministry 
within six months after fiscal-year end. The Ministry is responsible for tracking 
performance measures and is required to consolidate the information into an 
annual report to the PREA board for review and transmission to the Minister. 


By October 1, 2002, a third of the PREA recipients sampled had not submitted 
financial reports; more than half had not submitted performance reports for the 
previous fiscal year. Most of the reports received were submitted late. The 
Ministry did not ensure that all required reports were received on time nor was 
there evidence of adequate follow-up action. 


The Ministry did not adequately review the financial reports submitted, and the 
accuracy of many was questionable. For several of the reports reviewed, the 
cumulative value of the province’s contribution as reported by the recipients in 
successive periods exceeded the maximum provincial contribution under the 


funding agreement. The Ministry had not identified these discrepancies for 
follow-up. 
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Recipient institutions are responsible for tracking and reporting specific 
performance measures. The Ministry is responsible for consolidating performance 
information and for reporting annually on the program’s results. In addition to not 
ensuring that all required performance reports are received, the Ministry had not 
consolidated the information that was submitted and had not reported to the 
PREA Board and the Minister as required. 


The Auditor recommended that the Ministry ensure that all required financial and 
performance reports are received on a timely basis; verify that funds are being 
spent for the purposes intended, that the information submitted is accurate, and 
that project targets and milestones are being met; and analyze and consolidate the 
performance information reported by recipient institutions to assess the program’s 
accomplishments, and report this information annually to the program’s board and 
to the Minister as required.” 


Committee Hearings 


A January 2004 PREA board meeting included discussions on how to respond to 
the Auditor’s recommendations on monitoring the verification of funds and 
analyzing performance information. Ministry staff were developing revised 
procedures, based on input from the PREA board, for implementation by March 
2004. They would also be reviewing program reporting requirements to ensure 
that funds are used in keeping with payment plans, that information is accurate, 
and that targets and milestones are being met. The Ministry was also developing a 
system. to monitor and report on the program’s accomplishments by March 

2004. 


Monitoring Potential Conflicts of Interest 

Management Board has issued two directives that address potential conflict-of- 
interest situations: one for current and former public servants (Conflict of Interest 
and Post-Service Directive for Public Servants and Public Officials) and another 
for individuals appointed by the government (Government Appointees). 


The Minister relies on individuals from the research community to review funding 
proposals and sit on program boards and panels. Many advisory boards 
recommend to the Minister which research proposals should be funded. While 
board and panel members generally receive little or no remuneration, they may be 
involved in research proposals presented for funding or be shareholders in 
companies that are partnered with researchers submitting proposals. In addition, 
appointees may sit on several boards and panels. The Ministry has no procedures 
in place for monitoring potential conflicts of interest. 


Members of boards and panels who have an interest in a research proposal are 
expected to withdraw from any discussion of that proposal. The Auditor’s staff 
reviewed the board minutes of several programs and noted instances where 
members appropriately self-declared instances of potential conflict and withdrew 
from discussions on related proposals. They also noted instances where a conflict 
should have been declared, but there was no indication in the minutes that it had 
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been declared. The Ministry did not have sufficient information to determine 
whether all conflicts had been appropriately declared because board appointees 
and advisory panel members are not required to disclose potential or real conflicts 
of interest when they are appointed or if their circumstances later change. 


The Auditor recommended that the Ministry develop consistent conflict-of- 
interest policies that apply to all science and technology grant programs; develop 
standardized procedures for adequately monitoring potential conflicts of SS 
and inform the responsible Minister of all conflicts of interest as required. 


Committee Hearings 


The Ministry has endeavoured to address conflict-of-interest issues within all of 
its current programs, but acknowledged the Auditor's concern that this needed to 
be done with enhanced and systematic rigour. 


Steps had been taken to strengthen this focus by establishing a working committee 
to review existing provisions and develop consistent policies that apply to the 
decision and advisory processes in all transfer payment programs. Updated 
guidelines were expected to be prepared in time to annex to the new MOU with 
the Challenge Fund board. In the meantime, increased attention was being paid to 
the management of conflict-of-interest issues during board meetings.” 


Board members were said to be rarely in a situation where they do not declare a 
conflict-of-interest. Boards are meticulous about declarations. Because there is 
usually a formal record of proceedings, Ministry staff thought the Auditor’s staff 
may have observed instances that should have been minuted. They referred to the 
Challenge Fund board, saying it had adopted conflict of interest principles that a 
corporation would have in place.“ 


Because of the small number of experts in certain areas, conflict in reviewing 
applications and allocating money was considered natural. Conflict was defined in 
terms that relate to institutional and research project interests, and personal 
investments or interests in industrial partners. Once it is clear that a proposal 
under review is one in which a board or review panel member may have an 
interest, the conflict is declared and the individual leaves, only to return after a 
decision has been made. 


A large proportion of those involved in the decision-making process are from 
research-intensive universities. Close to 50 institutions have benefited from 
science and technology programs. Most projects are associated with half a dozen 


institutions, bodies from which board and review panel members are chosen.*° 


Following the hearings, the Ministry advised the Committee that it expected to 
have a conflict-of-interest policy in place by the end of May 2004.*° 
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4.4 Project Benefits 


Intellectual Property Rights 

Intellectual property (IP) rights represent the legal ownership resulting from 
research and academic activities. The owner of IP can exclude others from using 
it, and can transfer or sell it. One of the Ministry’s major objectives is to support 
job creation and economic growth, therefore grant recipients have a responsibility 
either to use IP in Ontario or license it for use here. 


The Auditor’s staff reviewed a number of programs to determine whether 
discoveries were benefiting the province. They found that there were no general 
guidelines for safeguarding the Ministry’s interests regarding IP rights. 


The Auditor recommended that the Ministry ensure compliance with program 
policies on IP rights; review existing policies and develop consistency among 
programs regarding the ownership of intellectual property; and formally assess the 
various programs’ success in meeting their objectives.*” 


Committee Hearings 


The Ministry planned to develop consistent principles with respect to IP rights 
across all science and technology programs. A new working committee will 
review policies on IP rights and determine the best way to optimize Ministry 
objectives as it works with an array of research institutions that have their own 
evolving IP rights policies and procedures. 


Criteria sensitive to the significant ongoing global debate about the role of IP 
policy in enabling early commercialization of research discoveries will be put in 
place to assess the programs' success in meeting these objectives. The approach 
taken in the development of IP policy options will be consistent with similar 
programs in competitor jurisdictions in Canada and North America. The target 
date for completion is the end of 2004.*8 


IP ownership for initiatives funded by the Ontario government tends to be 
determined by the recipient institution. Policies range from the traditional 
approach where an institution owns the IP to situations where more of the IP 1s 
owned by the principal investigator.” The latter is an international trend that sees 
the principal investigator work with an institution on protecting the academic 
pathway of conference papers. At the same time, the IP is developed and applied 
with assistance from the business arm of the institution in question. 


Much of the capital needed for investment in the front-end commercialization of 
projects may not be from Ontario, so there is potential for associated activities 
(e.g., R&D) to leave the province. Because the objective of government 
investment is to ensure that Ontario benefits from these projects, Ministry staff 
spoke of the need to examine ways of anchoring IP here. One such route would be 





* Ministry staff told the Committee that an area they continue to examine is multi-institutional 
projects. See Standing Committee on Public Accounts, Official Report of Debates (Hansard), p. P- 
165. 
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cluster development which tends to build local networks and attract the type of 


management expertise, capital and support that will facilitate initiatives staying 


iF: 5 
within an area. 


Committee Recommendation 


The Committee appreciates that research institutions across the province have 
their own evolving policies and procedures regarding intellectual property rights. 
It does, however, feel there is a need for provincial participation in this evolution, 
considering the investment the government makes in research activities at 
universities, colleges, hospitals, and other research institutions. 


The Committee therefore recommends that: 


5. The Ministry of Economic Development and Trade report to the 
Committee on the establishment, mandate and recommendations of 
the working committee reviewing intellectual property rights across 
the research institutions it funds. In particular, the Ministry should 
report on efforts to anchor these rights in Ontario. 


The Committee requests that the Ministry provide the Committee 
Clerk with a written response within 120 days of the tabling of this 
report in the Legislature. 


Industry Support 

The Auditor’s staff reviewed the cash and in-kind contributions made by industry 
to Ministry-funded projects. They found that the Ministry did not ensure recipient 
compliance with program requirements for industry participation and that it had 
not established consistent policies for assessing in-kind contributions. 


The Auditor recommended that the Ministry verify that the required commitment 
confirmation letters are received before funding research projects, consistently 
apply the criteria for proportionate program funding and document justification 
for any exceptions, and develop policies for the independent valuation of in-kind 
contributions.”! 


Committee Hearings 


The Ministry is refining its focus on private sector contributions for the Challenge 
Fund to more fully track and document the private sector partnership 
contributions to research programs. This will serve an accountability objective 
and will better document the role and involvement of the private sector in the 
research-based innovation and commercialization agenda. 


As part of this process, it will develop policies to confirm contributions and 
ensure that there is independent valuation of in-kind contributions of a designated 
material value. Ministry staff expected this to be completed by the end of 2004,” 
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Members were told that there were examples of approved projects having fallen 
apart because of misunderstandings about the availability of private sector funds 
and the nature of a private sector partnership. In these instances, remedial action 
had been taken to ensure that private sector arrangements were in place or a 
project was terminated.” : 


5. PROGRAM FINANCIAL AND ADMINISTRATIVE CONTROLS 


5.1 Program Administration 


At the time of the audit, there were approximately 50 staff in the Science and 
Technology Division. Annual direct operating expenses were about $7.8 million. 


The Auditor’s staff reviewed the Division’s financial and administrative controls. 
They concluded that controls over the funding of transfer-payment recipients 
needed to be improved. Their findings included the discovery that almost 30% of 
the Challenge Fund projects approved since 1997 (and approved to receive $128 
million) were inactive and without contracts. Several programs had provided 
funds in excess of recipients’ then current needs. Board members without the 
proper authority had hired an individual to work for the Challenge Fund. No 
information systems were in place to help Ministry staff manage transfer-payment 
programs. 


The Auditor recommended that the Ministry assess the continued merit of any 
approved research projects that are inactive and, where necessary, terminate 
funding commitments to inactive projects; review prepaid funding, so that 
payments are made to cover only current needs; clarify the roles and 
responsibilities of advisory board members and others involved in administering 
science and technology programs; and develop an information system to provide 
the Ministry’s staff with the information needed for effectively overseeing its 
transfer-payment pro grams.” 


Committee Hearings 


The Ministry is enhancing financial and administrative controls in order to 
achieve program efficiencies. Over the past few years, the number and size of 
science and technology programs have increased significantly. Because of the 
large number of complex programs, and in the context of incorporating the 
Auditor's recommendations, the Ministry is developing a number of initiatives to 
improve program delivery and monitoring. 


For example, it is reviewing program mandates and delivery mechanisms as well 
as the administrative processes for the Challenge Fund, the PREA, the Platinum 
Medal, the OCE program and the Performance Fund, among other science and 
technology programs. It is developing explicit and transparent procedures and 
policies, a suite of documents, in the areas of conflict of interest, IP and valuation 
processes for in-kind contributions. Finally, it is assessing the allocation of 
staffing and the introduction of enhanced technological solutions to allow for 
more effective program management.” : 
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5.2 Innovation Institute of Ontario 


ime | iSSi ingle- dministration of the 
At the time it requested permission to single source a 10. : 
Challenge Fund to the Innovation Institute of Ontario (0), the Ministry did not 
perform a detailed analysis of alternative delivery options and did not want to 
issue a request for proposals (RFP). 


Payments to the IIO are to be based on actual costs incurred and are to be 
negotiated every year. The audit found that neither the negotiations nor — 
submissions of annual operating plans and budgets were timely. The Ministry did 
not receive audited financial statements from the IIO. The Auditor’s staff 
reviewed unaudited statements for 2000/01 and 2001/02. They found that the 
salaries and benefits paid to the IIO for administering the Challenge Fund 
exceeded the salaries reported for the corporation as a whole in both fiscal years. 
It was also unclear how expenses were being managed between the IIO and the 


Trust. 


The Auditor recommended that the Ministry assess whether the expected benefits 
of outsourcing have been achieved; insist on receiving a budget and operating 
plans from the IIO before each fiscal year begins, instead of after the year has 
been completed; ensure that it receives audited financial statements from the IIO 
for use in assessing the appropriateness of fees charged for administering the 
Challenge Fund; and ensure that the detailed breakdown of the budget 
submissions correlates with the expense categories used in the financial 
statements and follow up on any discrepancies. 


Committee Hearings 


The Ministry pledged to continue to refine its contractual and working 
relationship with the IIO.°’ Ministry staff were then questioned about the 
administration contract between the IIO and the government. Members learned 
that the contract was worth $1.35 million in 2003/04. The amount varies from 
year to year depending on the arrangements negotiated, however, the contract has 
been managed by the same group since its inception. 


Even though an RFP is normally required for a contract of that amount, the 
government chose to pursue alternative service delivery. Outsourcing was 
approved by Management Board of Cabinet through a process which ensured 
there was rigour exercised in the way the money was to be spent.*® 


6. MEASURING AND REPORTING ON PROGRAM EFFECTIVENESS 


6.1 Program Planning 


The Science and Technology Division was found to lack a strategic plan with 
clearly stated goals to help it fulfill its contribution to the Ministry’s mandate to 
foster competitive businesses and a prosperous economy. An overall Strategic 
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plan would allow the Division to set parameters and consistent policies for 
existing programs and to guide the development of new programs. 


The Auditor recommended that the Ministry review all research programs and 
prepare a detailed strategic plan that sets specific goals and objectives for research 
in the province; and outline policies — such as conflict-of-interest rules, project 
selection criteria, and monitoring guidelines — that all programs must follow 
regardless of the delivery mechanism.”” 


Committee Hearings 


The Ministry was working on the elaboration of a more detailed strategic plan for 
its science and technology programs that will define the relationship between 
R&D activities and the Ministry's overall economic development mandate. The 
new strategy will be set out and incorporated into the Ministry's strategic results- 
based planning process that was underway at the time of the hearings. : 


A handout prepared by the Ministry and distributed at the hearings cited March 
2005 as the date by which the strategic plan would be submitted to the 
government for consideration. Policies applicable to science and technology 
programs would be finalized in advance.°’ Ministry staff were asked how ongoing 
concerns would be handled in the interim. The Ministry was hoping to beat some 
of the deadlines it had spoken about. Members also heard that science and 
technology programs would be among the first the government planned to 
examine under a new results-based budgeting exercise.” 


Since the hearings, the Committee has learned that the Ministry planned to have a 
draft science and technology strategic plan ready for stakeholder review by the 
end of June 2004.°° 


6.2 Effectiveness Reporting 


The government’s business planning process requires the Ministry to report to 
Management Board and publish a combined business plan and annual report. The 
Auditor’s staff reviewed the Ministry’s business planning process and found that 
it did not measure and report publicly on its achievement of promoting 
innovation, economic growth and job creation. 


Programs with transfer payment commitments of $10 million or more were 
reviewed by the Auditor’s staff to determine if they had clearly stated 
performance measures. Annual reports were often found to be out of date and 
many programs just reported lists of grants, success stories or activity-based 
measures. The Ontario Centres of Excellence had developed the most 
comprehensive performance measures and reporting system. Even though their 
measures indicated a degree of success, the Auditor was concerned that the 
Ministry had not established targets or benchmarks against which to measure 
these results. 


ESTE) ean ee Ce 


The Ontario Innovation Index was established in 2002 to measure Ontario's ' 
transition to a knowledge-based economy by comparing it with other jurisdictions 
and identifying areas in need of improvement. The ae reported that the 
indicators used do not isolate the impact of the Division’s programs. 


The Auditor recommended that the Ministry develop performance measures, 
targets, and benchmarks that reflect its accomplishments and contributions to the 
overall goals of promoting innovation, economic growth, and job creation, 
perform the necessary assessments to measure whether its initiatives are effective 
in achieving overall ministry goals; and report on the actual achievement of these 
measures, explaining any significant deviations from established targets and 


benchmarks. 


Committee Hearings 


The Ministry is committed to measuring the contribution its programs make in 
promoting the growth of high-paying jobs and globally competitive clusters in 
Ontario. In that context, it is taking steps to improve its accountability for the use 
of public funds by not only reviewing performance measures as part of its overall 
review of science and technology programs but by defining and refining 
meaningful measures which allow for gauging the true impact of its initiatives and 
investments.° 


The handout prepared by the Ministry included reference to an inter-jurisdictional 
review of performance measures for science and technology programs. The 
review would be consistent with “current global trends and ‘best-in-class’ 
approaches to support innovation, economic growth and job creation.” Ministry 
staff were asked what jurisdictions were deemed worthy of a ‘best-in-class’ 
designation. 


The state of Massachusetts, through government and associations, has been 
attentive to measurement and outcomes. The Ministry is looking at institutional 
and governmental developments with respect to measurement, performance 
indices and research support. 


Research funding programs in the United Kingdom reflect a focus on 
performance measurement and results, as do other government activities. The 
focus and quality of research in a number of institutions have been shaped over 
the years by a results-based approach to program administration and 
Management. 


Committee Recommendation 


The Committee recommends that: 


6. The Ministry of Economic Development and Trade report to the 
Committee on the progress being made on its inter-jurisdictional 
review of performance measures for science and technology 
programs, and on developing Ontario performance measures. 
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The Committee requests that the Ministry provide the Committee 
Clerk with a written response within 120 days of the tabling of this 
report in the Legislature. 


7. COMMITTEE RECOMMENDATIONS 


The Committee requests that the Ministry of Economic Development and Trade 
provide the Committee Clerk with a written response to the following 
recommendations within 120 calendar days of the tabling of this report, unless 
otherwise specified in a recommendation. 


1. The Ministry of Economic Development and Trade must give due 
consideration to oversight, compliance and governance implications 
before creating other bodies like the Ontario Innovation Trust. 


2. The Ministry of Economic Development and Trade report on the 
accountability measures contained within its memorandum of 
understanding with the Ontario Innovation Trust. The Committee requests 
that the Ministry provide the Committee Clerk with a written response 
within 30 days of the tabling of this report. 


3. The Ministry of Economic Development and Trade report to the 
Committee on its progress in improving the Premier’s Research 
Excellence Awards’ selection process procedures. 


4. The Ministry of Economic Development and Trade report on its efforts 
and those of the Innovation Institute of Ontario to update policies and 
procedures respecting financial and program reporting for Ontario 
Research Development Challenge Fund grants. 


5. The Ministry of Economic Development and Trade report to the 
Committee on the establishment, mandate and recommendations of the 
working committee reviewing intellectual property rights across the 
research institutions it funds. In particular, the Ministry should report on 
efforts to anchor these rights in Ontario. 


6. The Ministry of Economic Development and Trade report to the 
Committee on the progress being made on its inter-jurisdictional review of 
performance measures for science and technology programs, and on 
developing Ontario performance measures. 
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